When is a conclusion worth deriving? A relevance-based analysis of indeterminate relational problems

When is a conclusion worth deriving? We claim that a conclusion is worth deriving to the extent that it is relevant in the sense of relevance theory (Sperber & Wilson, 1995). To support this hypothesis, we experiment with "indeterminate relational problems" where we ask participants what, if anything, follows from premises such as A is taller than B, A is taller than C. With such problems, the indeterminate response that nothing follows is common, and we explain why. We distinguish several types of determinate conclusions and show that their rate is a function of their relevance. We argue that by appropriately changing the formulation of the premises, the relevance of determinate conclusions can be increased, and the rate of indeterminate responses thereby reduced. We contrast these relevance-based predictions with predictions based on linguistic congruence. Acknowledgments: We thank Richard Griggs, Maxwell Roberts, Carlos Santamaria, Walter Schaeken, Jean Baratgin, Ira Noveck and Deirdre Wilson for their helpful comments on earlier versions of this article. We also thank Jean-Marc Bernard for his help in statistical treatment. Jean-Baptiste Van der Henst is supported by a Marie Curie fellowship. When is a conclusion worth deriving? A relevance-based analysis of indeterminate relational problems From any set of premises, an infinity of deductively valid conclusions can be derived. For instance, from the single premise The cat is on the mat, it follows, by the rules of propositional calculus, that It is not the case that the cat is not on the mat or that The cat is on the mat or the dog is in the kitchen. However, in psychology of reasoning, there are many problems where participants, asked what, if anything, follows from some set of premises, are judged to have given the right answer when they say that nothing follows. Of course, neither the psychologist nor the participants are demonstrating illogicality in such cases. Instead, it is tacitly understood that the participants are being asked for a valid conclusion worth deriving (rather than for just any logically valid conclusion). But what, exactly is the content of this tacit understanding? What makes a conclusion worth deriving? Here we propose an answer to this question: a conclusion worth deriving is one that is relevant in the sense of relevance theory (Sperber & Wilson 1995). We test this hypothesis experimentally by analyzing participants' performance in indeterminate relational problems. Determinate and indeterminate relational problems Some of the inferences most investigated by researchers on reasoning are relational inferences (Burt, 1919; Piaget, 1921; Hunter, 1957; DeSoto, London & Handel, 1965; Huttenlocher; 1968; Clark, 1969a, b; Potts, 1972; Quinton & Fellows, 1975; Trabasso, Riley & Wilson, 1975; Newstead, Mantkelow & Evans, 1982; Byrne & Johnson-Laird, 1989; Carreiras & Santamaría, 1997; Roberts, 2000). So-called three-term series problems well illustrate this type of inference, for instance: Bill is better than Pete, Pete is better than John. Typically, participants have to infer a new relation not explicitly mentioned in the premises, which enables them to answer a question asked by the experimenter (What is the relation between Bill and John? Who is the best? ). Three-term series problems have been investigated in many studies (for a review, see Evans, Newstead & Byrne, 1993, chap.6) and the central issue has been how the premises are mentally represented. This has given rise to a debate between advocates of the analogical and linguistic approaches. According to the analogical approach (DeSoto, London, & Handel, 1965; Huttenlocher, 1968, information conveyed by the premises is integrated in the form of a "spatial array" or "mental model" (Johnson-Laird, 1983; Byrne & Johnson-Laird, 1989). Supporters of the linguistic approach, whose main advocate has been H.H. Clark (1969a; 1969b), claim instead that the representation of a premise is linguistic and basically corresponds to the abstract proposition describing a relation between a predicate and one or more entities (e.g. BETTER [Bill, Pete]). Psychologists have also investigated problems in which one or more relations were left indeterminate. For example the premises, Bill is better than Pete Bill is better than John, are traditionally said to be indeterminate because the relation between Pete and John is left unspecified. The question typically asked by experimenters may require knowledge of the relation between the two unrepeated terms, as in "Who is the worst?" (this question does not have a determinate answer) or not, as in "Who is the best? (this question has a determinate answer). Broadly, experimental results show that it is more difficult to deal with indeterminate problems than with determinate ones (Hayes-Roth & Hayes-Roth, 1975; Moeser & Tarrant, 1977; Warner & Griggs, 1980; Mani & Johnson-Laird, 1982; Byrne & Johnson-Laird, 1989). According to supporters of the analogical approach, this is because it is impossible to build a unique integrated representation from an indeterminate set of premises (Byrne & JohnsonLaird, 1989). Indeterminate and determinate problems have essentially been approached in the same way. The "representation question" has been the crucial one, and evidence from the study of the two types of problem has been taken to bear on exactly the same issues. The main difference between them has been seen as one of relative difficulty. On the other hand, the willingness of participants to spontaneously draw a conclusion from such problems has never been investigated. For this, the question should be not, for instance, "Is Pete better than John?" or "Who is the best?", but simply "What, if anything, follows from the premises?" The question "What if anything follows?" has indefinitely many logically valid answers, but none of them stands out as particularly compelling. Although no previous tests had been conducted, it was predictable that many participants would say that nothing follows. On the other hand, when participants do give positive responses, these provide clear evidence of how they interpret the task and what they see as a conclusion worth deriving. The importance of participants' interpretations of a psychological task, and the influence of pragmatic factors on this interpretation, are increasingly recognised in the psychology of reasoning (e.g. Politzer & Noveck, 1991; Evans, 1995; Hilton, 1995; Sperber, Cara & Girotto, 1995; Politzer & Macchi, 2000; Van der Henst, 1999; Thompson, 2000; Van der Henst, 2000; Noveck, 2001). Here, we want to show how, at least with indeterminate relational problems, a pragmatic approach helps predict and explain participants' responses. In particular, we attempt to alter the rate and specific contents of positive responses by manipulating the formulation of the problem. We are not directly addressing the more standard issues of how participants mentally represent the premises of relational problems, or what procedure they follow in drawing a conclusion. The issue we are addressing is that of participants' expectations and goals. Cognitive expectations, relevance, and the (in)determinacy of the premises The difference between determinate and indeterminate three-term problems is that, in indeterminate cases, it is impossible to infer a relation between two of the terms that was not already described in the premises. This may violate participants' expectations about the kind of conclusion they should be able to draw in the context of a reasoning task. But what exactly are these expectations? What, in general, makes a conclusion worth inferring? What kind of conclusion do participants expect the experimenter to expect them to draw? We claim that conclusions worth deriving are conclusions that are relevant. What is relevance? In relevance theory, relevance is seen as a property of inputs to cognitive processes (e.g. stimuli, utterances, mental representations). An input is relevant to an individual at a certain time if processing this input yields cognitive effects. Examples of cognitive effects are the revision of previous beliefs, or the derivation of contextual conclusions, that is, conclusions that follow from the input taken together with previously available information. Everything else being equal, the greater the cognitive effects achieved by processing an input, the greater its relevance. On the other hand, the greater the effort involved in processing an input, the lower its relevance. Everything else being equal, it is clearly conducive to greater cognitive efficiency to aim at greater relevance in the inputs one processes. (The assumption that human cognition is geared to maximizing relevance is called the "cognitive principle of relevance"). Let us illustrate. Suppose it is already known that only the youngest of Barbara, Pamela, and Jane has to stay at home. In this context, the set of premises {Barbara is older than Pamela, Pamela is older than Jane} is relevant because it yields the three contextual conclusions Jane has to stay at home, Barbara does not have to stay at home, Pamela does not have to stay at home. This first set of premises is more relevant in this context than another set {Barbara is older than Pamela, Barbara is older than Jane}, which yields only one of the three previous contextual conclusions: Barbara does not have to stay at home. On the other hand, the first set of premises is less relevant in the context than the single premise Jane is the youngest, which yields the same three contextual conclusions with less processing effort. The relevance of a piece of information is relevance to its user. In trying to produce relevant information for the use of an audience, or for oneself in the future, the effort to be minimized is that of the user, and this may typically involve some extra effort on the part of the producer of information. For instance, a speaker who knows that both Barbara and Pamela are older than Jane might make the effort of reformulating this knowledge, and say "Jane is the youngest", so as to produce an utterance that is optimally relevant to the audience in the context. Relevance theory does not invoke an absolute measure of mental effort or of cognitive effect, and it does not assume that such a measure is available to the spontaneous workings of the mind. What is assumed is that, not always but quite often, the actual or expected relevance of two inputs can be compared. Such comparisons help individuals allocate their cognitive resources. They also make it possible to manipulate the relevance factor in experimental research. How is relevance involved in reasoning experiments? Relevance is involved in reasoning experiments in three ways: 1) As in any act of communication, communicated information is automatically presented as relevant to the addressee (this is known, in relevance theory, as the "communicative principle of relevance"). In the experimental case, this means that participants are expected to treat the set of premises communicated to them by the experimenter as if they were relevant to them (generally with an element of pretence, as when reading fiction or engaging in pretend play, since participants have no real use for the pseudo-information that, say, Bill is better than Pete). The information may be immediately relevant to them if they have background knowledge in the context of which it carries cognitive effects. It may also be potentially relevant, in that it may provide premises that would be useful in future contingencies. 2) Just as in any ongoing discourse, where each new utterance is expected to be relevant in a context partly determined by the interpretation of the previous utterances, each new premise in a problem is expected to be relevant in the context of the other premises. 3) Participants expect their response to be relevant to the experimenter. These responses could not achieve relevance to the experimenter by informing her of a solution that she already knows, but they can by informing her about the participants' abilities, in particular their ability to derive conclusions relevant to themselves. Relevance to the experimenter is achieved in the same way as in answering an exam question (the examiner knows the answer; what she does not know, and what is relevant to her, is whether the examinee knows or is able to work out the answer). How does extracting conclusions from a set of premises contribute to its relevance? When participants are presented with a specific question (e.g. "Which of Pete, Bill and John is the best?"), the premises are typically relevant to them by allowing them to deduce the requested answer. However, when participants are asked "What, if anything, follows?", it is less clear why and, and in what way, they should go beyond restating the premises. Their answer might demonstrate that they have understood the potential relevance of the premises for further reasoning, but then a puzzle arises. How can deductively deriving a conclusion and adding it to, or substituting it for, an initial set of premises yield a more relevant point of departure for further reasoning, given that nothing can be derived from this conclusion that wasn't already derivable from the initial premises? Here is the answer. A set of premises with some deductively derived conclusion added cannot be more relevant than the initial set on the effect side, but it can be more relevant on the effort side. If the initial set of premises is expected to be relevant in some context where it carries certain cognitive effects, then the augmented set (or, in some cases, just the conclusion) might carry the same effects in that context, but at a lower cost in terms of effort. In fact, the prior deduction of some specific conclusion may be a preliminary and effort-costly necessary step towards deriving these cognitive effects. In that case, adding the deductive conclusion to (or, in some cases, substituting it for) the initial set of premises increases its expected relevance. It can be seen as preparatory work that improves the individual's readiness to make use of the premises in the context of further information. We frequently encounter information which we think is likely to prove useful in the future. We then retain this information, and often process it in such a way as to optimize its potential usefulness. Suppose, for instance, that you arrive in a holiday resort where you plan to spend a month with your family. You learn that there are three doctors in the resort, Smith, Jones, and Williams. You also learn the following two pieces of information: {Smith is a better doctor than Jones, Jones is a better doctor than Williams}. At the time, you don't need a doctor, but you might in the future, and would then want to visit the best doctor in town. So the information is potentially relevant to you. You might just store the two pieces of information above, but from a cognitive point of view it would be more efficient to draw the conclusion: Smith is the best doctor straight away. By drawing this conclusion now, you prepare for future circumstances in which you would need a doctor. By adding this conclusion to the two initial premises, you are left with a set of premises for future inference with a greater expected relevance, since its exploitation will require fewer inferential steps. Moreover, if you expect not to need information about the other two doctors, it may be sufficient to remember just the conclusion Smith is the best doctor, replacing the initial twopremise set with the single derived conclusion, thus reducing the memory load. What relevant conclusion should participants expect to be able to infer? Expecting each premise to be relevant in the context of the others means expecting some conclusion to be derivable from the premises taken together that was not derivable from the premises taken separately. Combining expectations of relevance to oneself (point 1 above) with the expectation that the premises will be relevant to one another (point 2), participants should expect to be able to derive from the premises taken together some conclusion that increases the relevance of the information contained in these premises by reducing the effort needed to achieve future cognitive effects. Moreover, combining this with the expectation of producing a response that will be relevant to the experimenter (point 3), participants should expect to be able to produce a response that demonstrates to the experimenter their ability to derive from the premises taken together a conclusion potentially relevant to themselves in the way just explained. Experiment To investigate participants' conception of a conclusion worth deriving, we asked what, if anything, followed from simple three-term relational problems. We compared their responses on the one hand to determinate vs. indeterminate problems, and on the other hand, and in much greater detail, to two versions of indeterminate problems. Types of possible conclusions and their relevance Whether or not participants produce a conclusion depends, we have claimed, on the expected relevance of the conclusions they can draw. In the case of determinate problems, such as {A is taller than B, B is taller than C}, the conclusion that can be drawn about the relationship between the two unrepeated terms, A and C in our example, stands out as clear and potentially relevant. We therefore expect most participants to produce such conclusions. In the case of indeterminate problems, no obviously relevant conclusion stands out, and so we must look at the different types of possible conclusions. We take as an example the set of premises {A is taller than B, A is taller than C}. We can distinguish three types, with the third dividing into two sub-types: (1) Conclusions based on a single premise, for instance: A is taller than some other item A is taller than B or 2+2=4 If B is taller than A, then A is taller than B (2) Conclusions connecting the two premises, for instance: A is taller than B and A is taller than C A is taller than B or A is taller than C (3) Conclusions integrating the two premises: (3a) Single-subject conclusions (i.e. with the repeated term A as subject), for instance: A is taller than B and C A is the tallest (3b) Double-subject conclusions (i.e. with the conjunction of the unrepeated terms B and C as subject), for instance: B and C are shorter than A B and C are the shortest Everyone would presumably predict that participants are unlikely to produce conclusions of type (1), but it may still be worth spelling out why. It cannot be just that these conclusions contain no new information, since this is true of all deductive conclusions. It cannot be that these conclusions are immediately obvious. Many (for instance: If B is taller than A, then A is taller than B) are not. We justify this prediction by noting that conclusions of type (1), which are based on only one premise, violate the expectation that both premises are relevant, and that each is relevant in the context of the other. Everybody would presumably predict that participants are unlikely to produce conclusions of type (2), but again, why? To begin with, we would argue that processing effort spent in conjoining (and even more, disjoining) two given premises in a conclusion is mis-directed. Typically, conjoined propositions have to be separated into two atomic propositions which can then serve, jointly or separately, as premises for further inferences. By contrast, there are few occasions on which it is useful to conjoin or disjoin premises in order to perform some real-life inference (for an argument that this is never necessary, and that inferences based on introduction rules are trivial and irrelevant, see Sperber & Wilson 1995: 95-103). In any case, even if some relevance could be found for conclusions of type (2), those of type (3) are definitely more relevant, and we have argued that participants try to produce the most relevant conclusion possible. Single-subject conclusions are more relevant than double-subject conclusions With indeterminate problems, conclusions of type (3) are potentially relevant. Given a context of independently obtained information, they might serve as useful inputs for further inferences. There are significant differences between conclusions of type (3a) (single-subject) and (3b) (double-subject) which affect their expected relevance. Both types come in either comparative form (A is taller than B and C, B and C are shorter than A) or superlative form (A is the tallest, B and C are the shortest). In fact, since the superlative form must be interpreted with respect to the set of three items A, B, and C that is, the tallest or the shortest means the tallest of A, B, and C or the shortest of A, B, and C -, these conclusions, whether of type (3a) or (3b), and whether of comparative or superlative form, have the same informational content. However, this content is not presented in the same way in conclusions of types (3a) and (3b). In particular, in one case the grammatical subject and topic consists of a single item (A), and in the other, it consists of the conjunction of two items (B and C). In other terms, the information is "about" A in one case, and "about" B and C in the other. It is a commonplace that the linguistic structure of a statement indicates what the statement is about and affects the way it is attended, processed, and remembered. Everything else being equal (and, in particular, the informational content being the same), single-topic conclusions have a greater expected relevance than multiple-topic conclusions. This has to do with the way new information is likely to be obtained and processed in human cognition generally. Our identification of individual items and categories is not random. It tends to pick out autonomous things or sets of things that have relatively stable interlocking properties and coherent and predictable behavior. This tends to maximize the inferable consequences of recognizing something as a given item or as belonging to a given category. Human categorization is geared to representing information in a way that maximizes the cognitive effects that can be derived from it, and minimizes the effort needed to derive these effects; in other words, it is geared to the maximization of relevance (this, by the way, is an application of the cognitive principle of relevance to the issue of categorization). For instance, it is quite generally more useful from a cognitive point of view to individuate as entities spatio-temporally continuous and autonomous objects (e.g. the dog Fido) rather than discontinuous spatial or temporal parts of objects (e.g. the entity made up of the dog Fido and the cat Julius). Similarly, it is more cognitively useful to group in the same category entities with similar and interrelated properties rather than less coherent ensembles (e.g. a category of cats is more useful than a category of animals or vehicles whose name in English begins with a "d"). Thus, more often than not, we pick out, we store, we use as premises, and we communicate information about single items or categories. Of course, almost any piece of information relates several items or categories, but in verbally or mentally representing the information, we typically treat it as information about a specific item or category, for instance by making this the subject or topic of an utterance. When we are presented with information about two or more items, we are likely to break it down for storage and further processing. For instance, if you learn that John and Billy are friends of Martha, you are likely to remember and use in the future as distinct pieces of information that John is a friend of Martha, that Billy is a friend of Martha, and/or that Martha has two friends, John and Billy. Information is stored under the mental "entries", "concepts", or "files" that we have for recognized entities or categories. You are much more likely to have one mental entry for John and one for Billy, than one for both John and Billy (unless they together make up a category such as the Blues Brothers). In ordinary circumstances, this is an optimal way of storing information, since you are more likely, in the future, to obtain new information about any given individual than about any given pair of individuals. Given these considerations, conclusions of type (3a), with a single subject and topic, are potentially more relevant than those of type (3b), with a double subject and topic. Again, there is no difference in actual informational content. The difference is only in the form, which affects the effort needed to store, retrieve and use the conclusion in a context of further information. The difference in relevance between the two types of conclusions is strictly on the effort side. Conclusions of types (1) and (2) are not relevant at all. Conclusions of type (3a) are more relevant than those of type (3b). We therefore predict, first, that participants who do give a determinate answer will produce only conclusions of type (3), and, second, that they will produce more single-subject conclusions (type 3a) than double-subject conclusions (type 3b). This second prediction is far from obvious, especially if it is extended from the specific type of indeterminate problem we have used as an example to another, symmetric, kind of indeterminate problem. As we will show, a possible and, at first blush, reasonable approach, based on Clark's (1969a,b) principle of congruence would make the same predictions as ours for one type of problem and exactly opposite predictions for the other. Let us contrast, then, two types of indeterminate problems. In one type, the two premises of the indeterminate problem have the same subject, as in the example we have already used {A is taller than B, A is taller than C}. In a second type, the two premises have different subjects, as in {B is taller than A, C is taller than A}. 1 With both same-subject premises and different-subject premises, it is possible to derive single-subject or double-subject conclusions. Let us represent the four possibilities in table form (see table 1):